Many low-cost sensors may spatially or electronically undersample an image. This results in an aliased image in which the high frequency components are folded into the low frequency components in the image. Consequently, subtle/detail information (high frequency components) are lost in these images. An image/signal processing method, called super-resolution image reconstruction, can increase image resolution without changing the design of the optics and the detectors. In other words, super-resolution image reconstruction can produce high-resolution images by using the existing low-cost imaging devices from a sequence (or a few snapshots) of low resolution images. The emphasis of the super-resolution image reconstruction algorithm is to de-alias the undersampled images to obtain an alias-free or, as identified in the literature, a super-resolved image.
When undersampled images have sub-pixel shifts between successive frames, they represent different information from the same scene. Therefore, the information that is contained in all undersampled images can be combined to obtain an alias-free (high-resolution) image. Super-resolution image reconstruction from multiple snapshots provides far more detail information than any interpolated image from a single snapshot.
There are three major steps in super-resolution image reconstruction methods. They are:                A) Acquiring a sequence of images from the same scene with sub-pixel shifts (fraction pixel displacements) among the images;        B) Estimating the sub-pixel (fraction pixel) shift or displacements among the images and        C) Reconstructing the high-resolution image.        
In the first step, there are two types of methods used to acquire low resolution images with sub-pixel shifts among them. One method is to have a controlled pixel displacement (see: (1) U.S. Pat. No. 6,642,497, “System for improving image resolution via sensor rotation,” Nov. 4, 2003, J. Apostolopoulos and F. Kitson; (2) U.S. Pat. No. 6,344,893, “Super-resolving imaging system,” Feb. 5, 2002, D. Mendlovic et. al.; (3) U.S. Pat. No. 6,483,952, “Super resolution methods for electro-optical systems,” Nov. 19, 2002, D. D. Gregory et. al.). In this method, a special sensor or scanner (hardware) is designed to capture multiple images in a known pattern, where each image is captured by displacing the sensor in a known distance that is not a multiple of a pixel, but rather is a multiple of a pixel plus a known fraction of a pixel. Another method is to have a non-controlled pixel displacement, e.g. natural jitter. This method is more cost effective and more practical. For example, an imager is carried by a moving platform in many applications. In a rescue mission, the camera may be carried by a helicopter, or a moving vehicle. In military reconnaissance situations, the camera may be carried by a person, an unmanned ground vehicle (UGV), or an unmanned aerial vehicle (UAV). Sub-pixel shifts need to be accurately estimated for natural jittering imagers.
The second step is to estimate the sub-pixel shift or fraction pixel displacements. There are many methods that are addressed in the literature. Frame-to-frame motion detection based on gradient decent methods are most used (see: (1) U.S. Pat. No. 5,767,987, “Method and apparatus for combining multiple image scans for enhanced resolution,” Jun. 16, 1998, G. J. Wolff and R. J. Van Steenkiste; (2) U.S. Pat. No. 6,650,704, “Method of producing a high quality, high resolution image from a sequence of low quality, low resolution images that are undersampled and subject to jitter,” Nov. 18, 2003, R. S. Carlson, J. L. Arnold, and V. G. Feldmus; (3) U.S. Pat. No. 6,285,804, “Resolution improvement from multiple images of a scene containing motion at fractional pixel values,” Sep. 4, 2001, R. J. Crinon and M. I. Sezan; (4) U.S. Pat. No. 6,349,154, “Method and arrangement for creating a high-resolution still picture,” Feb. 19, 2002, R. P. Kleihorst). One of the variations of these methods is to estimate the velocity vector field measurement based on spatio-temporal image derivative (see U.S. Pat. No. 6,023,535, “Methods and systems for reproducing a high resolution image from sample data,” Feb. 8, 2000, S. Aoki). Most of these methods need to calculate matrix inversion or use iterative methods to calculate the motion vectors. Bergen (see U.S. Pat. No. 6,208,765, “method and apparatus for improving image resolution,” Mar. 27, 2001, J. R. Bergen) teaches a method to use warp information to obtain sub-pixel displacement. Stone et al (see U.S. Pat. No. 6,628,845, “Method for subpixel registration of images,” Sep. 30, 2003, H. S. Stone, M. T. Orchard, E-C Chang, and S. Martucci) teaches another method that is to estimate the phase difference between two images to obtain sub-pixel shifts. In this method, the minimum least square solution has to be obtained to find the linear Fourier phase relationship between two images.
The third step in super-resolution image reconstruction is to reconstruct the high-resolution image. Many methods have been proposed to solve the problem. These can be divided into two categories: directive non-uniform interpolation and non-directive inverse processing. Among directive interpolation methods, U.S. Pat. No. 5,767,987 teaches a method in which the reference image is interpolated to produce an image of higher density samples (mixels). Each of the remaining pixel image scans is aligned with the interpolated prototype. The mixel values of the prototype are iteratively adjusted by minimizing an error metric representing the difference between the pixel values computed from the prototype and the corresponding pixel values of each of the low resolution pixel image scans. Other directive interpolation methods are described using spatial interpolation at the inter-pixel position (see U.S. Pat. No. 6,285,804), using warping procedure (see U.S. Pat. No. 6,208,765), using weighted interpolation (see U.S. Pat. No. 6,023,535), and implementing the interpolation by convolving with a shifted-fractional kernel (polynomial kernel) (see U.S. Pat. No. 6,650,704).
In non-directive inverse processing methods, an observation model is formulated to relate the original high resolution image to the observed low resolution images. The solution of high resolution image is often solved by inverse matrix or iterative procedure. The observation model can be formulated in spatial domain or frequency domain. The corresponding inverse methods are implemented in both domains. For spatial domain inverse method, initial guess of a high resolution image is obtained first (see M. Irani and S. Peleg, “Improving resolution by image registration,” CVGIP: Graphical Models and Image Processing, vol. 53, pp. 231-239, May 1991). Then an imaging formation processing is simulated to obtain a set of low resolution images. The differences between the simulated and original low resolution images are used to improve the initial guess of the high resolution image. In this method, the initial guess of the high resolution image is crucial to the algorithm convergence. For frequency domain inverse methods, the relationship between low-resolution images and the high resolution image is demonstrated in the frequency domain (see R. Y Tsai and T. S. Huang, “Multiple frame image restoration and registration,” in Advances in Computer Vision and Image Processing, Greenwich, Conn.: JAI Press Inc., 1984, pp. 317-339), in which the relation between the continuous Fourier Transform (CFT) of an original high resolution image and the discrete Fourier Transform (DFT) of observed aliased low resolution images is formulated based on the shifting property of the Fourier transform.
When there is prior knowledge about the high resolution image and the statistical information of noise can be defined, the inverse processing method can provide stabilized (regularized) estimates. These methods are called regularized super-resolution approaches (see: (1) B.C. Tom and A. K. Katsaggelos, “Reconstruction of a high-resolution image by simultaneous registration, restoration, and interpolation of low-resolution images,” Proc. of 1995 IEEE Int. Conf. Image Processing, vol. 2, Washington, DC, October 1995, pp. 539-542; (2) R. R. Schulz and R. L. Stevenson, “Extraction of high-resolution frames from video sequences,” IEEE Trans. Image Processing, vol. 5, pp. 996-1011, June 1996; (3) R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP registration and high-resolution image estimation using a sequence of undersampled images,” IEEE Trans. Image Processing, vol. 6, pp. 1621-1633, December 1997).
In consideration of the problems detailed above and the discrepancies enumerated in the partial solutions thereto, an object of the present method is to produce high-resolution images by using a sequence of low-resolution images from existing low-cost imaging devices.
Another object of the present method is to produce high-resolution images for unknown sub-pixel shifts among the low-resolution images by estimating sub-pixel accuracy shifts (displacements).
Another object of the present method is to produce high-resolution images that eliminate alias from the low-resolution images.
In order to attain the objectives described above, according to an aspect of the present invention, there is provided a method of super-resolving images whereby super-resolution images are reconstructed from sequences of, typically lower resolution, aliased imagery.